Recognition of weed in a natural environment

ABSTRACT

A method for identifying of a type of weed in a natural environment may be provided. The method comprises receiving a digital image of the weed in an early growth stage together with related metadata, contouring areas of leaves and determining whether the weed is a monocotyledon or a dicotyledon. Furthermore, the method comprises determining a growth stage and determining the type of weed identified by a weed name and a probability of the correctness of the determination using at least one out of a plurality of the received metadata and a plurality of metadata determined during the contouring, during the determining whether the weed is a monocotyledon or a dicotyledon, and during the determination of the growth stage of the weeds as input parameters to a set of classifiers using a storage comprising names of types of weeds together with a plurality of sets of metadata per weed type.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a national stage application (under 35 U.S.C. § 371)of PCT/EP2017/060751, filed May 5, 2017, which claims benefit ofEuropean Application Nos. 16169416.1, filed May 12, 2016, and16182582.3, filed Aug. 3, 2016, all of which are incorporated herein byreference in their entirety.

FIELD OF THE INVENTION

The invention relates generally to weed identification, and morespecifically to a method for identifying a type of weed in a naturalenvironment. The invention relates further to an identification systemfor an identification of a type of weed in a natural environment, and acomputer program product.

BACKGROUND

Farming is currently undergoing its next development stage. After anindustrialization of many farming processes and a series of automationinitiatives, the introduction of high-tech tools in the farming processcontinues. Adding to this context, the phenomenon of the Internet ofThings (IoT) does not at agriculture.

Because farmers are more than interested in increasing their harvestingyields as well as reducing their costs, many efforts are put intomeasures to enhance farming efficiency. A considerable influence on theyield is given by reducing factors like pests, illnesses and weeds inthe fields. In order to fight weeds, farmers consequently useherbicides, but such treatment in general involves high efforts andcosts. So far, when applying herbicides, farmers need to rely on theirexperience and individual knowledge to identify weeds. This expertisemight not be met by all farming personnel, so any help when deciding onhow to fight existing weeds is highly appreciated. In addition, acorrect identification of weeds, and thus choosing the right herbicidesin the correct amount, is instrumental to saving resources and furtherprotect the environment.

Following this, there is a need for technical assistance for identifyingweeds in a farm field without the manual process of picking individualleaves of a potential weed plant out in the field and trying to identifythe type of weed in a back-office using some literature. It is thereforean objective of the current application to provide a method and a systemfor identifying weeds in a natural environment among cultivated crop.

SUMMARY

This need may be addressed by a method for identifying of a type of weedin a natural environment, a recognition system for an identification ofa type of weed in a natural environment, and a computer program product,according to the independent claims.

According to a first aspect of the invention, a method for identifying atype of weed based on a digital image of the weed in a naturalenvironment may be provided. The method may comprise receiving a digitalimage comprising data representing a weed in an early growth stage inthe natural environment among cultivated crop. Additionally, metadatarelating to the digital image may be received.

The method may further comprise contouring areas with a predefined colorand texture specification in an RGB color model within the digital imagethereby building at least one contoured area comprising pixels of theweed within a boundary contour, and determining whether the weed is amonocotyledon or a dicotyledon.

Furthermore, the method may comprise determining a growth stage of theweed by isolating at least a single leaf of the weed by determining ajoint center of the contoured area, and determining the number of leaveswithin the contoured area.

A further feature of the method may comprise determining the type ofweed identified by a weed name and a probability of the correctness ofthe determination using at least one out of a plurality of the receivedmetadata, and a plurality of derived metadata determined during thecontouring, during the determining whether the weed is a monocotyledonor a dicotyledon, and during the determination of the growth stage ofthe weeds as input parameters to a set of classifiers. The set ofclassifiers may have access to a storage comprising names of types ofweeds together with a plurality of sets of metadata per weed type.

According to another aspect of the present invention an identificationsystem for an identification of a type of weed based on a digital imageof the weed in a natural environment may be provided. The identificationsystem may comprise a receiving unit adapted for receiving a digitalimage comprising data representing a weed in an early growth stage inthe natural environment among cultivated crop, and metadata relating tothe digital image, a contouring unit adapted for contouring areas with apredefined color and texture specification in an RGB color model withinthe digital image building at least one contoured area comprising pixelsof the weed within a boundary, and a first classifier unit adapted fordetermining whether the weed is a monocotyledon or a dicotyledon.

The identification system may further comprise a determination unitadapted for determining a growth stage of the weed by isolating at leasta single leaf of the weed by determining a joint center of the contouredarea, and determining the number of leaves within the contoured area.

Additionally, the identification system may comprise a second classifierunit comprising a set of second classifiers adapted for determining thetype of weed identified by a weed name and a probability of thecorrectness of the determination using at least one out of a pluralityof the received metadata, and a plurality of metadata determined by thecontouring unit, the first classifier unit, and the determination unitfor determining the growth stage of the weed as input parameters. Thesecond classifiers may have access to a storage system comprising namesof types of weeds together with a plurality of sets of metadata per weedtype.

It may be noted that the identification system may be implemented aspart of a private computing center or a shared computing center, like acloud computing center. The digital image may be taken by a digitalcamera in the farm field and received by the identification system forfurther processing. The image capturing device or digital camera may beoperated manually by the farmer, a consultant or any other person.Alternatively, the camera may be installed on a flying drone, which maybe operated autonomously or which may be controlled remotely.

Furthermore, embodiments may take the form of a related computer programproduct, accessible from a computer-usable or computer-readable mediumproviding program code for use, by or in connection with a computer orany instruction execution system. For the purpose of this description, acomputer-usable or computer-readable medium may be any apparatus thatmay contain means for storing, communicating, propagating ortransporting the program for use, by or in a connection with theinstruction execution system, apparatus, or device.

The proposed method for recognizing of a type of weed in a naturalenvironment may offer a couple of advantages and technical effects:

Automatically detecting a type of weed and determining the correct weedname is of high value for farmers in the field. Weeds in a crop fieldmay change from season to season, from weather condition to weathercondition and from one year to another year. In addition, the phenotypicappearance of weeds may change based on environmental conditions suchthat also specialists may not be able to identify a type of weed in areliable manner. Especially during early growth stages, it is difficultto recognize small differences in the appearance of a specific weed.Using a camera, e.g., from a smartphone, taking a picture, performing apreanalysis within the smartphone and/or sending the picture to anidentification system may provide significant value for the farmer. Hemay decide instantly which type of herbicide to use in order to fightthe developing weed among his crop. It may also help to apply theherbicide only onto those areas where the weed has grown. In order tofight weeds only in spots of a farming field, additional zoninginformation of the distribution of the weeds in the field may berequired.

The task of capturing an image of a potential weed may also be performedby noneducated farmers, support personnel or temporary staff without anyexperience in recognizing weed. This may save valuable time and money inthe farming process. It may be possible to reduce the image informationof the digital image to a useful minimum. Together with the usage of ahigh performance, highly sophisticated weed identification system in theform of a set of classifiers or a neural network system, the farmer maybe enabled to determine the name of the weed only using his smartphonemore or less instantaneously. The usage of metadata as input values forthe classifiers determining the weed name together with the probabilityof one or more weed types and sending this information back to thecapturing smartphone may allow for a sophisticated interaction betweenthe farmer and the backend system. It also allows determining that weedname with a high probability and to advise the farmer how to proceed.Moreover, it may also be possible to implement the classifier technologyin high-performance multi-core based smartphones or dedicated weedanalysis systems.

Furthermore, this whole process of capturing and receiving the image ofthe weed and sending back the weed name—with a probability rating—may beperformed within a very short amount of time, e.g., seconds. No specialequipment, like special cameras, may be required in the field becausesmartphones are omnipresent today. Thus, the costs for the recognitionprocess may also be kept comparably low in comparison to traditionaltechnologies because a large plurality of simple smartphones may use theidentification system in a cloud computing manner.

Finally, the weed identification process, performed by the highlysophisticated identification system may be available also to anunexperienced farmer allowing him to fight weeds and increase hisfarming yield.

Furthermore, the proposed method and system allows an identification,determination and/or recognition of weed in a natural environment andnot only in the glass house under artificial conditions. This may proveto be an invaluable advantage to farmers working in the field becauseeverything can be achieved live and under realistic daily workingconditions. Recognizing weed in a natural environment has proven to bemuch harder than under glass house conditions because the number ofvariables is significantly higher. In a glass house there may beconstant light, no direct sunshine or single light source related strongshadowing, no light angle variations related to time of the day, leadingto changing and unpredictable reflections and color appearance, nocloud, no fog or haze or varying soil conditions, no varying wetnessconditions, no wind induced plant movement, no insect related leafdamage, just to name a few parameters why a recognition in a naturalenvironment cannot be compared with a weed recognition in a glass house.Thus, all of the above-mentioned circumstances vary under naturalconditions, which may represent a significant difference to partiallyknown technologies for image recognition which rely typically on optimaland artificial environmental conditions.

In the following, additional embodiments of the proposed method foridentifying of a type of weed in a natural environment will bedescribed. It may be noted that the embodiments may be implemented inform of the method or in form of the related system.

According to one preferred embodiment of the method, the predefinedcolor specification may relate to a color range of weed in a naturalenvironment, featuring the complete visible range and in particularfocus on the green color range of wavelength of, e.g., 490 to 575 nm.This may reduce “image noise” from background information also capturedas part of the digital image. Also the background of the captured weedmay comprise single or small groups of green pixels falling into thiswavelength range. The contouring may eliminate this background greeninformation spots. The allowable wavelength range (e.g., 490 to 575 nm)may be stored in a table, a database or another suitable data format.

The texture specification may relate to leaf veins, characteristic formsof leaf segments, specific patterning and color distribution, microhairs on the surface atop of the leaf and on the edge of the leaf. Allof these and additional characteristic textures may not explicitly beparameterized but belong to the “learnt” context of the trainedclassifier correlation function(s).

According to an additionally preferred embodiment of the method, thecontouring the areas with the predefined color and texture specificationmay be performed by determining for every combined color of the digitalimage whether a combination of its color components may match one of aplurality predefined color combinations. Typically, image detectorsfunction using the known RGB color model (red-green-blue) utilizing 3sub-pixels for every complete pixel, wherein one sub-pixel is used forone of the 3 basic colors red, green and blue. The so specified colorspace boundaries with predefined color information and distribution ofintensities within the RGB model of the sub-colors pixel information maybe stored as reference. This way, a fast comparison between each pixelof the digital image and the stored reference color information may beperformed by which a pixel of the captures digital image may be selectedas part of the to-be-contoured area or not.

According to one advantageous embodiment of the method, contouring ofareas with the predefined color specification may be performedadditionally by a determination of1. w _(i) =F(p _(i) ,p _(i,j)), wherein  (Eq. 1)

w_(i)=1 or 0 indicating that pixel i may belong to weed or not. This maybe performed in addition to the more straight forward comparison againstallowed color information for a single pixel. F may be a functioncalculating a probability for weed, respectively non-weed based on colorattributes of p_(i) and all of p_(j), p_(i)=pixel i, and p_(i,j)=pixelsj surrounding the pixel i. The number of pixels counted as surroundingpixel i may vary. E.g., only one ring or surrounding pixels may be used;this may be 8 pixels p_(i,j). Moreover, a next ring of pixels p_(i,j)surrounding the first ring may also be considered; this second ring maycomprise additional 16 pixels p_(i,j). Additional pixel rings may beconsidered and different rings may be multiplied with decreasingweighing factors the more distant a pixel ring is from pixel i. This mayenable a more thorough determination whether a pixel of the captureddigital image should count as weed pixel or not.

According to one permissive embodiment of the method, the early growthstage may be defined by a BBCH code from 10 to 39. Using the BBCH code(the international accepted code from Biologische Bundesanstalt,Bundessortenamt and CHemische Industrie in Germany) may help to achievecomparable results in an early growth stage of weed. Typically, weedwith a growth stage according to BBCH code below 10, e.g., first leavesemerging, may not be recognizable even by an expert. Weed with a BBCHcode larger or equal 40 may be too much grown up to fight itsuccessfully with herbicides. However, also weed with a BBCH code below10 may be recognized with a lower probability. Increased spectralanalysis of the color information of the contoured area may beinstrumental for achieving this.

According to one useful embodiment of the method, the determiningwhether the weed is a monocotyledon or a dicotyledon may compriseapplying a classifier which has been trained to distinguish betweenmonocotyledon and dicotyledon weeds resulting in a binary monocot-dicotidentifier value. Classifiers like neural network classifiers may bewell suited for a differentiation between monocotyledon and dicotyledonweed types if the classifiers are well trained with training images.Weed of either the monocotyledon type or the dicotyledon type comprisecharacteristic features allowing a good differentiation between the twotypes of weed. The monocot-dicot identifier value may be treated as aderived metadata.

According to one advantageous embodiment of the method, derived metadatamay comprise at least one selected out of the group comprising a binarymonocot-dicot identifier value, a number of leaves determined, and thedetermined growth stage. These metadata may be metadata derived from thereceived digital image. They may be differentiated from the receivedmetadata although both types of metadata may finally be used asparameters to determine the type of weed.

According to a further advantageous embodiment of the method, thereceived metadata may comprise at least one out of the group comprisingglobal positioning system data (GPS) values of a capturing location ofthe digital image—in particular for identifying a country or region—acalendar date—in particular for an identification of a season andpotentially related weather data—topography data related to the globalpositioning data system data values—in particular an identification ofthe height above sea level and or geological data—acceleration data of acapturing device during a moment of capturing the digital image, a tiltangle of the capturing device during capturing the digital image, andthe camera type the digital image was captured with. Some of thesemetadata may be received together with the digital image from, e.g., asmartphone or alternatively from one or more databases from a server.They may also be received for another computing system, e.g., a cloudcomputing center providing weather data or topology data based in GPSand date/time data.

According to an additionally preferred embodiment of the method, eachmetadata value—received and/or derived—may have a predefined weighingvalue which may also be used as input parameter to the set ofclassifiers. These parameters may be used for a fine tuning of theproposed method and system. The parameters may be learned automaticallycomparing data for different seasons or they may be set manuallyreflecting made experiences with the system by the farmers and managers.

According to a further preferred embodiment of the method, thedetermination of the type of weed identified by its weed name may alsocomprise comparing the name of the determined weed name with a weedprobability map which may map probability occurrences of weeds forgeographic locations and seasonal data and recalculating the probabilityof the correctness of the determination based on the comparing the nameof the determined weed name with a weed probability map. This additionaladjustment may contribute to an even more precise determination of thetype of weed. It may, e.g., exclude a wrongly identified type of weedbecause the probability to determine that type of weed at the given timeof the year (season) and/or geographic region may be relativelyunlikely.

It may also be mentioned that other meta data value may be included inthe determination process like the type of weed grown in the actualseason and seasons before including the related types of weed detectedduring these earlier seasons on the actual field or on other fields inthe same or similar crop growing regions. This type of additionalinformation may also be refected in the probability map.

Furthermore, the method may comprise sending the identified weed namesand a related probability of the correctness of the determination forthe weed names having the three highest probability values. The wirelesssending may—in case a human being may have operated the digital camerafor capturing the digital image—be directed to the operator of thecamera, and more precisely to a smartphone of the operator. If—on theother hand—a digital camera may be installed on an automatic vehiclelike a drone or an automatic wheel based field robot, the names andprobabilities may be sent to a farmer's smartphone or another mobiledevice of the farmer wirelessly including the GPS coordinates where theweed was found. A special software application may then render thelocation of the found weed on a geographical map of the farm.

Furthermore, embodiments may take the form of a related computer programproduct, accessible from a computer-usable or computer-readable mediumproviding program code for use, by or in connection with a computer orany instruction execution system, like a smartphone. For the purpose ofthis description, a computer-usable or computer-readable medium may beany apparatus that may contain means for storing, communicating,propagating or transporting the program for use, by or in a connectionwith the instruction execution system, apparatus, or device.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

It should be noted that aspects of the invention are described withreference to different subject-matters. In particular, some embodimentsare described with reference to method type claims whereas otherembodiments have been described with reference to device type claims.However, a skilled person of the art will gather from the above and thefollowing description that, unless otherwise notified, in addition toany combination of features belonging to one type of subject-matter,also any combination between features relating to differentsubject-matters, in particular, between features of the method typeclaims, and features of the device type claims, is considered as to bedisclosed within this document.

The aspects defined above and further aspects of the present inventionare apparent from the examples of embodiments being describedhereinafter. They are explained with reference to the examples ofembodiments, but to which the invention is not limited.

Preferred embodiments of the invention will be described, by way ofexample only, and with reference to the following drawings:

FIG. 1 shows a block diagram of an embodiment of the inventive methodfor recognizing of a type of weed in a natural environment.

FIG. 2 shows a block diagram of an embodiment of a capturing of adigital image of a dicotyledon.

FIG. 3 shows a block diagram of an embodiment of a capturing a digitalimage of a monocotyledon.

FIG. 4 shows an embodiment of steps of the area contouring process.

FIGS. 5a, b illustrate a concept of including further rings of pixels inthe contouring process.

FIG. 6 shows an embodiment of an image of weed with four leaves and ajoint center.

FIG. 7 shows a block diagram of an embodiment of the weed identificationsystem.

FIG. 8 shows an embodiment of a computing system comprising theidentification system.

DETAILED DESCRIPTION

In the context of this description, the following conventions, termsand/or expressions may be used:

The term ‘identifying’, in particular ‘identifying a type of weed’ or‘recognizing a type of weed’ or also ‘determining’ may denote anautomated machine-based determination or recognition process for anidentification of a specific type of weed starting from a receiveddigital image from a digital camera, pre-processing of the digitalimage, deriving metadata during the processing of the digital image anduse these by, e.g., neural network based classifiers for a probabilitybased analysis of the image data, and finally an identification of oneor more type(s) of weed.

The term ‘weed’ may denote an unwanted plant of any species that mayquickly adapt to almost any environment. Here, the expression weed mayalso be related to plants among crop or cultivated or economic plantsthat are useful in the sense of harvesting fruits of grown up seed. Weedmay disturb or negatively impact the process of the crop growth anddecrease the yield of an agriculture field.

The term ‘natural environment’ may—in the context of plants like crop orweed—denote that the plants grow in a field or on land which may beexposed to natural weather and environmental conditions, like humidityand direct or indirect sun light and other weather phenomena. Hence, anatural environment excludes artificial environments like glass housesor other non-natural growing environments for plants. Such unnaturalenvironments with controlled conditions may artificially excludenumerous influence factors, which make the recognition process muchharder or—in many cases—impossible. This feature may prove to be anadvantage of the currently proposed method and system because it may bemuch easier to differentiate between crop and weed in an artificiallycontrolled environment. Under extremely controlled conditions it may bemuch easier to recognize a specific type of weed because a large numberof types of weeds may be excluded upfront given the specific andcontrolled conditions of, e.g., a glass house.

The term ‘early growth stage’ may denote a stage of a plant in which theplant, in particular the weed, may not have grown to an adult stage.Very early growth stages may be difficult to recognize anyway. It hasbeen shown that the usage of the ‘BBCH code’ may be useful whendescribing growth stages of plants, e.g., weed. The abbreviation BBCHstands officially for “Biologische Bundesanstalt, Bundessortenamt andCHemische Industrie” and describes phenological growth stages of aplant. The code goes from 00 to 99. A BBCH code of 10 to 19 representsdifferent early development stadiums of leaves. The principal growthstage 2 includes BBCH codes 20 to 29 and is about formation of sideshoots/tillering. The principal growth stage 3 (BBCH codes 30 to 39)comprises stem elongation/shoot development (main shoot). Thus, focusingon weed with BBCH codes between 10 and 39 may represent a good focus onweed in an early growth stage.

The term ‘contouring’ may denote a process of a determination of acontour of a certain area or surface having common or similar colorand/or textual characteristics of, e.g., weed in a digital picture. Eachleaf of plants, in particular weed, has a natural boundary or outer edgeor edges of the leaf. The process of contouring captures, recognizesand/or determines these edges such that inside the related contouredarea all or almost all pixels of the weed may be included.

The term ‘color and texture specification’ may denote digitalinformation about pixels in a digital image according to a color model,e.g., the RGB color model (red, green, blue). However, other colormodels may be used, like HSV (hue, saturation, value), HSL (hue,saturation, lightness/luminance). It is well known in the industry thatmost color model information from one color model may be transformed toanother color model by a mathematical matrix operation. Different colormodels may have different advantages like most natural colorrepresentations, best suitable for digital processing, optimally adaptedfor grayscale processing, best suited for edge recognition and thus forcontouring, and so on.

The term ‘RGB color model’ may denote the well-known additive colormodel in which red, green, and blue lights are added together in variousways to reproduce a broad array of colors. The name of the model comesfrom the initials of the three additive primary colors, red, green, andblue. The main purpose of the RGB color model is for the sensing,representation, and display of images in electronic systems, such astelevisions and computers, though it has also been used in conventionalphotography. Before the electronic age, the RGB color model already hada solid theory behind it, based in human perception of colors. RGB is adevice-dependent color model: different devices may detect or reproducea given RGB value differently since the color elements (such asphosphors or dyes) and their response to the individual R, G, and Blevels vary from manufacturer to manufacturer, or even in the samedevice over time. Thus, an RGB value may not define the same coloracross devices without some kind of color management and mathematicaltransformations.

The term ‘classifier’ or ‘classifier correlation function’, and inparticular ‘trained classifier correlation function’ may denote one ormore mathematical functions allowing to measure a similarity of featuresbetween one or more sections of a captured image and a set of referenceimage data by which the classifiers may have been trained. The featureparametrization of a correlation function may not be programmed manuallybut may be trained, i.e., learned using datasets with a known pluralityof input attributes as well as the desired result. A skilled person willknow various types of correlation approaches.

Actually, this approach may also be used for the texture specificationas well as the monocotyledon/dicotyledon differentiation and/or the typeof weed determination. No parameters may be specified explicitly butautomatically derived during the training sessions of the classifiercorrelation function(s).

Several types of classifiers are known and may be used for the inventiveconcept. Abstractly, a classifier—e.g., implemented as an algorithm—mapsinput data to a predefined category. Classifiers are typically used inmachine learning. They are trained using a defined set of training datawhich shall generate a known outcome, representing a sort of supervisedlearning. One example for a classifier is a linear classifier in whichthe predicted category is the one with the highest score. This type ofscore function is known as a linear predictor function and has thefollowing general form:Score (X _(i) ,k)=β_(k) *X _(i),

X_(i) is the feature vector for instance i, β_(k) is the vector ofweights corresponding to category k, and score (X_(i), k) is the scoreassociated with assigning instance i to category k. Feature vector maybe binary, categorical, ordinal, integer-valued or real-valued.Classifiers may also work as binary or multi-class classifier.

Another type of classifier is a bio-inspired neural network classifierworking with simulated neurons receiving a set of input values andsynaptic elements building a link between an output of one neuron and aninput of another neuron. From manually annotated images (trainingimages), a set of features of the plants is extracted and used to trainthe neural network (supervised learning). After training, the same setof features is extracted for each new image (test image) and the neuralnetwork uses these features to classify the plant in the image.

The term ‘weed probability map’ may denote a geo-based map denotingprobabilities for finding a weed at an geographical position determinedby geo-coordinates on the earth's surface and related altitude valuescombined with seasonal data as well as historic data about times andintensity of having found a specific weed at the location in question.Thus, in simple words, the weed probability map may be instrumental inanswering the question: How probable is it to find a specific type ofweed at a given location at a given time in the year.

In the following, a detailed description of the figures will be given.All instructions in the figures are schematic. Firstly, a block diagramof an embodiment of the inventive method for identifying of a type ofweed in a natural environment is given. Afterwards, further embodimentsas well as embodiments of the identification system for a recognition ofa type of weed in a natural environment will be described.

FIG. 1 shows a block diagram of an embodiment of the method 100 foridentifying a type of weed based on a digital image taken in a naturalenvironment. The method comprises receiving, 102, a digital image. Thedigital image may be taken by an operator of a digital camera, e.g., afarmer in the field or a camera on an autonomous vehicle. The farmer mayuse the digital camera of a smartphone for taking the digital image andsending it to the identification system. The digital image received via,e.g., the smartphone by the identification system may comprise datarepresenting a weed in an early growth stage in the natural environmentamong cultivated crop as well as metadata relating to the digital image.The metadata may comprise GPS data of the location of the digital image,i.e., the weed together with a height value above zero, etc. of thelocation of taking the digital image.

The method comprises further contouring, 104, areas with a predefinedcolor and texture specification in an RGB color model within the digitalimage building at least one contoured area comprising pixels of the weedwithin a boundary contour, in particular a boundary line. There may beat least one weed plant or eventually more. With other words, greenplant areas are detected that are different to a background, e.g., thesoil of the field. Furthermore, the method may comprise determining,106, whether the weed is a monocotyledon or a dicotyledon. These twokinds of weeds may be identified relatively easy because the number ofembryonic leaves or cotyledons may be determined using a trainedclassifier. Alternatively, an input may be made by the operator of thecamera whether weed of the digital image is a monocotyledon or adicotyledon. Even untrained farmers or operators may differentiate thesetwo types of plants.

Additionally, the method may comprise determining, 108, a growth stageof the weed by isolating, 110, at least a single leaf of the weed bydetermining a joint center of the contoured area and determining, 112,the number of leaves within the contoured area. This may be performedusing an image processing function taken from an advanced digital imageprocessing library, like, e.g., the Halcon library from a company namedMVTec. The term ‘isolating’ may denote a process of differentiatingpixels relating to the weed from pixels of the background of the digitalimage. Thus, it may be a logical isolation such that the weed may betreated as one logical object of the data structure of the digitalimage.

The actual segmentation into single leaves may be performed in thefollowing way: The digital image is converted from the RGB color spaceinto the HSV (Hue Saturation Value) color space and segmented usingthresholding, resulting in a binary segmentation. In order to getindividual leaves, firstly a distance transformation is applied insidethe foreground region of the binary segmentation. After that, the localmaxima of the distance map are used as seeds for a subsequent watershedsegmentation, which separates individual leaves from each other.

A further step of the method is the determining, 114, the type of weedidentified by a weed name and a probability of the correctness of thedetermination using at least one out of a plurality of the receivedmetadata and derived from at least one out of a plurality ofcharacterizing and/or derived metadata determined during the contouring,during the determining whether the weed is a monocotyledon or adicotyledon, and during the determination of the growth stage of theweeds. These metadata are used as input parameters to a set ofclassifiers. The at least one classifier of the set may have access to astorage system, i.e., a database comprising names of types ofweeds—e.g., several hundred—together with a plurality of sets ofmetadata per weed type.

The classifiers may a selection out of neural network classifiers,linear classifiers and/or sample based identifiers. Additionally,library or historic data may be used during the identification process,like historic weed identification data from earlier seasons at the sameor a comparable time, special weather conditions like actual and/orhistoric weather conditions, etc.

Last but not least the method may comprise sending (not shown) resultsof the determination—e.g., the three most probable results with therelated probability of the determination—to, e.g., back to thesmartphone the digital image was received from.

FIG. 2 shows a plant, in particular weed and an image capturing device202. This may be a camera of a smartphone (not shown). The camera 202may have an objective and a certain opening angle 204. The captureddigital image may capture the leaf(s) 204 of the weed, here adicotyledon. The stem 208 may, e.g., be covered by the one or moreleaves 206. In the cross-section also the earth surface 210 and the root212 of the weed is shown. In an optimal case, the camera's image plane202 may be completely parallel to a longitudinal extension of the leaves206, i.e., along the earth surface. However, smaller deviationsindicated by the angle α may be acceptable.

It may also be noted that the digital image may not only have capturedmore than one leaf of a weed, but also more than one plant of potentialweed. However, ideally, each captured image would only have one weedplant or leaf for an easier recognition process. The captured digitalimage of the weed may be transmitted, e.g., by wireless technology fromthe camera 202 to a data center comprising the identification system foran identification of weed in a natural environment. In one embodiment,the camera 202 may be a component of a smartphone. In anotherembodiment, the camera itself may be equipped with wireless sendingand/or receiving technology. The camera may also be mounted on a droneor a moving earthbased robot in the field.

FIG. 3 shows a comparable scenario for monocotyledon 302, e.g. grass.Capturing a digital image from above a monocotyledon 302 may not revealenough information for a classification and determination of the type ofweed. Therefore, monocotyledon 302 may be digitally photographed fromthe side of the plant, or the weed may be extracted from the soil andput flat on the soil. In that case, a capturing of the weed may beperformed as discussed in the case of FIG. 2. Otherwise, if themonocotyledon roots are still in the soil, the optical capturing plane203 of the digital camera 202 should be vertical to the earth surface.Also here, smaller deviations indicated by the angle α may beacceptable.

FIG. 4a, 4b, 4c show the process of contouring. FIG. 4a shows aprojection of a weed leaf 402 on an image sensor 203 and/or a respectivematrix of pixels. Each small square 403 may represent one pixel. Askilled person will acknowledge that, for the purpose of explanation, avery coarse-grained matrix is used here. Typically, the camera may havea resolution of several million pixels. FIG. 4b shows the result of adigitization. Now, the smooth edges of the leaf 402 are gone and theoriginal analog shape is digitized (404). In a next step—FIG. 4c —thecontour of the potential leaf may be extracted or isolated from thedigital image. Everything inside the contour 406 may be counted as aleaf or weed pixel.

FIG. 5a shows details of the determination process whether a certainpixel may belong to a weed leaf or not. In the simplest case, the colorvalue of a simple pixel p_(i) 502 is used as determination criterion.The related color range may comprise, e.g., wavelengths of 495 to 575nm. However, sometimes such a simple differentiation may not besufficient. Therefore, pixels surrounding the pixel 502 in question arealso taken into account. FIG. 5 shows a first “ring” of additionalpixels directly surrounding pixel 502. Equation (1) may be used to alsouse the information related to pixel p_(i) 502.

FIG. 5b shows in addition to the first “ring” 504 of pixels surroundingpixel 502 a second “ring” 506 of surrounding pixels. The influence orweighing of the decision whether pixel 502 is determined to be a leafpixel or not, may decrease the further away a certain pixel may be fromthe pixel 502 in question. Further rings of pixels p_(i,j) may be usedand may become variables of the function F. The number of rings used maybe a function of the total resolution of the received image.

FIG. 6 shows a digital image 602 of an example of weed comprising fourleaves 604, 606, 608 and 610. Based on this, using a classifier oranother suitable image recognition library component, the method allowsa determination of a growth stage of the weed by isolating orrecognizing at least one single leaf of the weed. For this, a jointcenter 612 may be determined as part of the contoured area comprisingthe, e.g., four leaves 604, 606, 608, and 610. Using the same classifieror the same or another suitable image recognition library component alsothe number of leaves within the contoured area may be determined. Thisallows for a determination of the growth stage of the weed according tothe BBCH code. The result may be stored as a derived metadata.

FIG. 7 shows a block diagram of an embodiment of the identificationsystem 700 for an identification of a type of weed based on a digitalimage in a natural environment. The identification system comprises areceiving unit 702 adapted for receiving a digital image comprising datarepresenting a weed in an early growth stage in the natural environmentamong cultivated crop. Additionally, the metadata are received relatingto the digital image. The identification system 700 also comprises acontouring unit 704 adapted for contouring areas with a predefined colorand texture specification in an RGB color model within the digital imagebuilding at least one contoured area comprising pixels of the weedwithin a boundary. A first classifier unit 706 is adapted fordetermining whether the weed is a monocotyledon or a dicotyledon. Agrowth stage determination unit 708 is adapted for determining a growthstage of the weed by isolating, i.e. recognizing at least a single leafof the weed by determining a joint center of the contoured area anddetermining the number of leaves within the contoured area.

A set of second classifier units 712 is adapted for determining the typeof weed identified by a weed name and a probability of the correctnessof the determination using at least one out of a plurality of thereceived metadata, and a plurality of metadata determined by thecontouring unit, by the determining the first classifier unit, and thedetermination unit as input parameters, the set of the second classifierunits having access to a storage comprising names of types of weedstogether with a plurality of sets of metadata per weed type.

A summing unit may combine the determination of the individualclassifiers in case the classifiers work in parallel. Alternatively, oneor more classifiers could also work in series, such that an output ofone classifier may be an input to a subsequent classifier. A network ofartificial neurons with artificial synaptic elements may be used forthat.

Embodiments of the invention may be implemented together with virtuallyany type of computer—in particular a smartphone—regardless of theplatform being suitable for storing and/or executing program code. FIG.8 shows, as an example, a computing system 800 suitable for executingprogram code related to the proposed method. Beside smartphones, alsoother mobile devices with a camera, a processor for executing programcode and a transceiver may be suited for the implementation of theproposed method and/or related recognition system.

The computing system 800 is only one example of a suitable computersystem and is not intended to suggest any limitation as to the scope ofuse or functionality of embodiments of the invention described herein.Regardless, computer system 800 is capable of being implemented and/orperforming any of the functionality set forth hereinabove. In thecomputer system 800, there are components, which are operational withnumerous other general purpose or special purpose computing systemenvironments or configurations. Examples of well-known computingsystems, environments, and/or configurations that may be suitable foruse with computer system 800 include, but are not limited to, tabletcomputers, hand-held or laptop devices, multiprocessor systems,microprocessor-based systems, programmable consumer electronics,smartphones, and digital camera with spare computing capacity thatinclude any of the above systems or devices, and the like. Computersystem 800 may be described in the general context of computersystem-executable instructions, such as program modules, being executedby a computer system 800. Generally, program modules may includeroutines, programs, objects, components, logic, data structures, and soon that perform particular tasks or implement particular abstract datatypes.

As shown in the figure, computer system 800 is shown in the form of ageneral purpose computing device. The components of computer system 800may include, but are not limited to, one or more processors orprocessing units 802, a system memory 804, and a bus 818 that couplesvarious system components including system memory 804 to the processor802. Computer system 800 typically includes a variety of computer systemreadable media. Such media may be any available media that is accessibleby computer system 800, and it includes both, volatile and non-volatilemedia, removable and non-removable media.

The system memory 804 may include computer system readable media in theform of volatile memory, such as random access memory (RAM) and/or cachememory. Computer system 800 may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia. By way of example only, storage system 812 may be provided forreading from and writing to a non-removable storage chip. Storage mediacan be connected to bus 806 by one or more data media interfaces. Aswill be further depicted and described below, memory 804 may include atleast one program product having a set (e.g., at least one) of programmodules that are configured to carry out the functions of embodiments ofthe invention.

A program/utility, having a set (at least one) of program modules, maybe stored in memory 804 by way of example, and not limitation, as wellas an operating system, one or more application programs, other programmodules, and program data. Program modules may generally carry out thefunctions and/or methodologies of embodiments of the invention asdescribed herein.

The computer system 800 may also communicate with one or more externaldevices such as a keyboard, a pointing device, a display 820, etc.;these devices may be combined in a touch-screen that enable a user tointeract with computer system 800; and/or any devices (e.g., networkcard, modem, etc.) that enable computer system 800 to communicate withone or more other computing devices. Such communication can occur viainput/output (I/O) interfaces. Still yet, computer system 800 maycommunicate with one or more networks such as a local area network(LAN), a general wide area network (WAN), and/or a mobile public network(e.g., the Internet) via network adapter 822. As depicted, networkadapter 814 may communicate with the other components of computer system800 via bus 818. It should be understood that although not shown, otherhardware and/or software components could be used in conjunction withcomputer system 800. Examples, include, but are not limited to:microcode, device drivers, redundant processing units, etc.

Additionally, the identification system 700 for an identification of atype of weed in a natural environment may be attached to the bus system818.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinaryskills in the art without departing from the scope and spirit of thedescribed embodiments. The terminology used herein was chosen to bestexplain the principles of the embodiments, the practical application ortechnical improvement over technologies found in the marketplace, or toenable others of ordinary skills in the art to understand theembodiments disclosed herein.

The present invention may be embodied as a system, a method, and/or acomputer program product. The computer program product may include acomputer readable storage medium (or media) having computer readableprogram instructions thereon for causing a processor to carry outaspects of the present invention.

The medium may be an electronic, magnetic, optical, electromagnetic,infrared or a semi-conductor system for a propagation medium, like e.g.,solid state memory, a random access memory (RAM), a read-only memory(ROM).

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device such as an EPROM, or any suitablecombination of the foregoing. A computer readable storage medium, asused herein, is not to be construed as being transitory signals per se,such as radio waves or other freely propagating electromagnetic waves,electromagnetic waves propagating through a waveguide or othertransmission media (e.g., light pulses passing through a fiber-opticcable), or electrical signals transmitted through a wire.

Computer readable program instructions described herein can bedownloaded to the respective computing devices, e.g. as a smartphone appfrom a service provider via a mobile network connection.

Computer readable program instructions for carrying out operations ofthe present invention may be any machine dependent or machineindependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including anobject-oriented programming language such as C++, Java or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the computerdevice. In some embodiments, electronic circuitry including, forexample, programmable logic circuitry, field-programmable gate arrays(FPGA), or programmable logic arrays (PLA) may execute the computerreadable program instructions by utilizing state information of thecomputer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus', and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus', or anotherdevice to cause a series of operational steps to be performed on thecomputer, other programmable apparatus or other device to produce acomputer implemented process, such that the instructions which executeon the computer, other programmable apparatus', or another deviceimplement the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowcharts and/or block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or act or carry out combinations of special purpose hardwareand computer instructions.

The corresponding structures, materials, acts, and equivalents of allmeans or steps plus function elements in the claims below are intendedto include any structure, material, or act for performing the functionin combination with other claimed elements, as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skills in the artwithout departing from the scope and spirit of the invention. Theembodiments are chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skills in the art to understand the invention forvarious embodiments with various modifications, as are suited to theparticular use contemplated.

The invention claimed is:
 1. A method for identifying a type of weedbased on a digital image of the weed in a natural environment, themethod comprising: receiving a digital image comprising datarepresenting a weed in an early growth stage in the natural environmentamong cultivated crop, and metadata relating to the digital image,wherein the early growth stage is defined by a BBCH code from 10 to 39,contouring areas with a predefined color and texture specification in anRGB color model within the digital image building at least one contouredarea comprising pixels of the weed within a boundary contour,determining whether the weed is a monocotyledon or a dicotyledondetermining the type of weed identified by one or more weed names andcorresponding probabilities of a correctness of the determination usingat least one out of a plurality of the received metadata, and at leastone out of a plurality of derived metadata determined during thecontouring, and during the determining whether the weed is amonocotyledon or a dicotyledon, as input parameters to a set ofclassifiers having access to a storage system comprising names of typesof weeds together with a plurality of sets of metadata per weed type,wherein the one or more weed names are compared with a weed probabilitymap which maps probability occurrences of weeds for geographic locationsand seasonal data to exclude wrongly identified names of weeds from thedetermination.
 2. The method according to claim 1, further comprises:determining a growth stage of the weed by isolating at least a singleleaf of the weed by determining a joint center of the contoured area,and determining a number of leaves within the contoured area, anddetermining the type of weed identified by the one or more weed namesand corresponding probabilities of the correctness of the determinationusing derived metadata determined during the determination of the growthstage of the weeds.
 3. The method according to claim 1, wherein thepredefined color specification relates to a color range of plants in anatural environment, in particular to a green color range of wavelength490 to 575 nm.
 4. The method according to claim 1, wherein thecontouring areas with the predefined color and texture specification isperformed by—determining for every pixel of the digital image whether acombination of its RGB color components matches one of a plurality ofpredefined color combinations.
 5. The method according to claim 1,wherein the contouring of areas with the predefined color specificationis performed additionally by a determination ofwi=F(pi,pij) wherein wi=1 or 0 indicating RGB that pixel i belongs toweed or not, F is a function calculating a probability for weed/non-weedbased on color attributes of pi and all of pj, pi=pixel i, pi,j=combinedpixels j surrounding the pixel i.
 6. The method according to claim 1,wherein the determining whether the weed is a monocotyledon or adicotyledon comprises applying a classifier which has been trained todistinguish between monocotyledon and dicotyledon weeds resulting in abinary monocot-dicot identifier value.
 7. The method according to claim1, wherein derived metadata comprise at least one selected out of agroup comprising binary monocot-dicot identifier value, a number ofleaves determined, and a development growth stage.
 8. The methodaccording to claim 1, wherein the received metadata comprises at leastone out of a group comprising global positioning system data values of acapturing location of the digital image, a calendar date, topographydata related to the global positioning data system data values,acceleration data of a capturing device during a moment of capturing thedigital image, a tilt angle of the capturing device during capturing thedigital image, and a camera type the digital image was captured with. 9.The method according to claim 1, wherein each metadata value has apredefined weighing value which is also used as input parameter to theset of classifiers.
 10. The method according to claim 1, wherein thedetermination of the type of weed identified by the one or more weednames also comprises recalculating the probabilities of the correctnessof the determination based on the comparing the name of the determinedone or more weed names with the weed probability map.
 11. The methodaccording to claim 1, also comprising sending the weed names and arelated probability of the correctness of the determination for the weednames having the three highest probability values to a device by whichthe digital image was taken.
 12. An identification system foridentifying a type of a weed based on a digital image of the weed, thedigital image comprising data representing the weed in an early growthstage in a natural environment among cultivated crop, the identificationsystem comprising a receiving unit adapted for receiving the digitalimage, and metadata relating to the digital image, a contouring unitadapted for contouring areas with a predefined color and texturespecification in an RGB color model within the digital image building atleast one contoured area comprising pixels of the weed within aboundary, a first classifier unit adapted for determining whether theweed is a monocotyledon or a dicotyledon, a set of second classifierunits adapted for determining the type of weed identified by one or moreweed names and corresponding probabilities of a correctness of thedetermination using at least one out of a plurality of the receivedmetadata, and at least one out of a plurality of derived metadatadetermined by the contouring unit, and by the first classifier unit, asinput parameters, the set of the second classifier units having accessto a storage comprising names of types of weeds together with aplurality of sets of metadata per weed type, wherein the one or moreweed names are compared with a weed probability map which mapsprobability occurrences of weeds for geographic locations and seasonaldata to exclude wrongly identified names of weeds from thedetermination.
 13. The identification system according to claim 12,further comprises: a determination unit adapted for determining a growthstage of the weed by isolating at least a single leaf of the weed bydetermining a joint center of the contoured area, and determining anumber of leaves within the contoured area, and wherein the set ofsecond classifier units is adapted for determining the type of weedidentified by the one or more weed names and the correspondingprobabilities of the correctness of the determination using derivedmetadata determined by the determination unit.
 14. The identificationsystem according to claim 12, wherein the predefined color specificationrelates to a color range of plants in a natural environment, featuringthe complete visible range with a focus in a green color range ofwavelength 490 to 575 nm.
 15. The identification system according toclaim 12, also comprising a sending unit adapted for sending the weednames and a related probability of the correctness of the determinationfor the weed names having the three highest probability values.
 16. Anon-transitory computer-readable medium having program instructionsencoded thereon that, when executed by one or more computing devices,cause the computing devices to: receive a digital image, and metadatarelating to the digital image, contour areas with a predefined color andtexture specification in an RGB color model within the digital imagebuilding at least one contoured area comprising pixels of a weed withina boundary contour, determine whether the weed is a monocotyledon or adicotyledon, determine a type of weed identified by one or more weednames and corresponding probabilities of a correctness of thedetermination using at least one out of a plurality of the receivedmetadata, and at least one out of a plurality of derived metadatadetermined during the contouring, and during the determining whether theweed is a monocotyledon or a dicotyledon, as input parameters to a setof classifiers having access to a storage comprising names of types ofweeds together with a plurality of sets of metadata per weed type,wherein the one or more weed names are compared with a weed probabilitymap which maps probability occurrences of weeds for geographic locationsand seasonal data to exclude wrongly identified names of weeds from thedetermination.
 17. The non-transitory computer-readable medium accordingto claim 16, wherein said program instructions are executable by one ormore computing devices to cause said one or more computing devices todetermine a growth stage of the weed by isolating at least a single leafof the weed by determining a joint center of the contoured area, anddetermining a number of leaves within the contoured area, and determinethe type of weed identified by the one or more weed names and thecorresponding probabilities of the correctness of the determinationusing derived metadata determined during the determination of the growthstage of the weeds.